Add RocksDB native compaction policy task by bochencs · Pull Request #156 · FrontierCS/Frontier-CS

bochencs · 2026-06-17T10:06:10Z

Summary

Adds a Frontier-CS 2.0 systems task for optimizing leveled compaction selection in a pinned RocksDB v10.10.1 checkout. Recent LSM-tree research continues to study workload-aware compaction and write-amplification tradeoffs; this task evaluates whether agents can produce robust picker changes across hidden native workloads.

The judge builds the submitted patch against RocksDB, checks snapshots, range deletes, multiple column families, reopen behavior, and full iterator contents, then measures write, read, and space amplification together with trusted residual compaction debt.

Type of Change

Testing

$ PYTHONDONTWRITEBYTECODE=1 python3 -m pytest -q 2.0/problems/rocksdb_native_compaction_policy/test_evaluator.py
..........................                                               [100%]
26 passed in 0.06s

$ uvx ruff check 2.0/problems/rocksdb_native_compaction_policy/evaluator.py 2.0/problems/rocksdb_native_compaction_policy/test_evaluator.py 2.0/problems/rocksdb_native_compaction_policy/harbor/app/validate_solution_patch.py
All checks passed!

$ PYTHONPATH=adapters/frontier-cs-2.0/src uv run --no-sync python -m frontier_cs_2_0.main --source "$PWD" --output-dir /private/tmp/frontier-cs-rocksdb-submit-audit --task-ids rocksdb_native_compaction_policy --overwrite
  [OK] rocksdb_native_compaction_policy
Generating tasks -> /private/tmp/frontier-cs-rocksdb-submit-audit/

Done: 1 tasks generated

$ bash 2.0/problems/rocksdb_native_compaction_policy/docker/build_images.sh
Built frontiercs/rocksdb-native-compaction-judge:experimental-v10.10.1-task2 for the local platform

$ docker run --rm -e FRONTIER_SUBMISSION_ROLE=final -v "$PWD/2.0/problems/rocksdb_native_compaction_policy:/task:ro" frontiercs/rocksdb-native-compaction-judge:experimental-v10.10.1-task2 python3 /task/evaluator.py /task/reference.patch
{"message": "scored; band=measurable", "metrics": {"build_ok": 1, "case_count": 14, "changed_files_count": 1, "component_floor": 0.91722, "composite_gain": 1.01163, "empty_patch": 0, "improved_profile_count": 4, "intra_l0_delta": 0.0, "material_regression_count": 0, "max_runtime_ratio": 1.018, "max_seed_log_spread": 0.09865, "mean_seed_log_spread": 0.0251, "profile_count": 7, "required_improved_profile_count": 3, "robust_gain": 1.01094, "score_band": "measurable", "valid_patch": 1, "worst_case_gain": 0.98694}, "score": 3.3232668009898805, "score_unbounded": 3.3232668009898805}
3.323266800990 3.323266800990

GPT-5.5 difficulty check

Three independent Harbor trials used the full 10800-second agent budget and completed with valid, non-empty patches and all 14 final cases. None achieved a positive score.

Trial 1: score=0; score_unbounded=-2.8312507002; valid_patch=1; empty_patch=0; build_ok=1; case_count=14; robust_gain=0.99997; improved_profile_count=0.
Trial 2: score=0; score_unbounded=-2.7713949625; valid_patch=1; empty_patch=0; build_ok=1; case_count=14; robust_gain=1.00007; improved_profile_count=0.
Trial 3 (non-empty final submission): score=0; score_unbounded=-3.4048022133; valid_patch=1; empty_patch=0; build_ok=1; case_count=14; robust_gain=0.99895; improved_profile_count=2; material_regression_count=0.

Judge Image

The task includes the judge Dockerfile, pinned RocksDB commit and archive checksum, native harness, and build script. The config currently uses the local experimental tag below, following the existing custom-image task pattern:

bash 2.0/problems/rocksdb_native_compaction_policy/docker/build_images.sh

For the final image, would you prefer that I publish it under my GHCR namespace, or should it be hosted under the FrontierCS organization?

Checklist

Code follows the project structure and conventions
Self-review completed
Documentation updated (if applicable)

CI Validation (for new problems)

The reference patch is included as both reference.patch for the Harbor adapter and reference.cpp for the current C++ reference lookup.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds the Frontier-CS 2.0 “LSM compaction policy” problem package, including documentation, a local evaluator/harness, and a reference policy implementation.

Changes:

Introduce a baseline/reference policy (reference.py) and problem statement (readme).
Add a full offline evaluator and sandbox runner (evaluator.py) plus a convenience entrypoint script (evaluate.sh).
Add runtime/submission metadata for the task (config.yaml).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
2.0/problems/lsm_compaction_policy/reference.py	Adds a simple reference compaction policy used for baseline behavior.
2.0/problems/lsm_compaction_policy/readme	Documents the task, policy interface, observation schema, scoring, and restrictions.
2.0/problems/lsm_compaction_policy/evaluator.py	Implements the simulator, scoring, sandboxed policy runner, and case generation.
2.0/problems/lsm_compaction_policy/evaluate.sh	Adds a shell wrapper to run the evaluator against a solution path.
2.0/problems/lsm_compaction_policy/config.yaml	Declares environment/runtime and submission contract for the task.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

joyemang33 · 2026-06-19T05:46:48Z

Hi @bochencs, thanks for your contribution. Can you implement an LSM tree task in a database-native environment such as rocksDB? maybe refer our duckdb_e2e_optimization problems. I think RocksDB is where the most LSM tree research will implement their code

bochencs · 2026-06-19T06:36:10Z

in a database-native environment such as rocksDB?

@joyemang33 Thanks for the quick check! That make sense for me, let me work on it

joyemang33 · 2026-06-20T00:14:30Z

something might also be useful: https://arxiv.org/pdf/2501.16759v3

bochencs · 2026-07-12T01:54:23Z

@joyemang33 I have replaced the simulator with a native RocksDB task. I also read the paper you linked, although it focuses on joins rather than compaction, it reinforced that the benchmark should operate on a real RocksDB implementation.

The task pins RocksDB v10.10.1 at 4595a5e95ae8525c42e172a054435782b3479c57. Agents patch five compaction-picker/version files in /app/rocksdb and submit a Git patch. The judge incrementally rebuilds the candidate and runs it against unmodified RocksDB on paired deterministic workloads covering L0 pressure, overlap-heavy rewrites, scans, range deletes, snapshots, time-series access, and multiple column families.

Final evaluation uses two hidden seeds for each of seven workload families. Correctness is checked through reads, snapshots, background errors, reopen, and a complete iterator comparison. An unmodified binary then drains residual compaction work, so deferring work does not improve the score.

The reference patch completed all 14 final cases with score=3.3233, robust_gain=1.01094, 4/7 improved profiles, and no material profile regression. The evaluator has 26 passing tests, the harness compiles cleanly with -Wall -Wextra -Wpedantic, and the task passes CLI discovery, adapter generation, image build, and RocksDB formatting checks. I also ran three full-budget GPT-5.5 Harbor trials; all produced valid non-empty patches, built successfully, completed all 14 cases, and scored zero (robust_gain: 0.99997, 1.00007, 0.99895). Full commands and logs are in the updated PR description.

The PR includes the Dockerfile, harness, pinned source checksum, and build script. It currently uses the local tag frontiercs/rocksdb-native-compaction-judge:experimental-v10.10.1-task2. Does this native RocksDB scope match what you had in mind? For the final image, should I publish it under my GHCR namespace, or would you prefer it under the FrontierCS organization? Would like to seek further guidance about the image.

joyemang33 · 2026-07-12T02:01:50Z

@joyemang33 Thank u so much! @momoway will help you review the task

momoway · 2026-07-14T11:37:57Z

Hi @bochencs — this is a great task and definitely worth adding to Frontier-CS 2.0! I ran an additional full Harbor trial of it (codex + gpt-5.5) and hit two issues worth addressing before merge.

1. Invalid submissions outscore valid ones on score_unbounded. Invalid submissions currently return score_unbounded = 0.0, while valid below-floor submissions get negative unbounded scores (−5.0 to −2.8 in my trial). A stall-guard-invalid submission therefore looks strictly better than every valid attempt. This sends a misleading signal to the agent, and the adapter's best-artifact selection also picks the invalid patch as "best" and re-runs it at final time in place of the agent's actual final patch. I'd suggest either not returning score_unbounded for invalid results, or returning a very negative sentinel (e.g. -1e6).

2. The score scale has no discrimination. The reference solution scores 3.32/100, and all agent trials so far (your three plus mine) score exactly 0. Since the reference is strong and clearly hard for current models to reach, I'd calibrate it so the reference lands at a high score — e.g. tune the target so it comes out around 50 or 80 — so the task can actually separate models instead of collapsing everything to 0.

…cale Address review feedback on FrontierCS#156: - _invalid() now returns score_unbounded = -1e6 so Harbor's (score, score_unbounded) best-artifact selection can never prefer an invalid submission over a valid or empty one; the empty no-op patch keeps (0.0, 0.0) as the documented neutral baseline. - TARGET_ROBUST_GAIN 1.20 -> 1.017: the reference patch (robust_gain ~1.011 on the final suite) lands near 50 instead of 3.32. The 1.005 noise deadband and every validity/breadth/regression gate are unchanged, so the zero-vs-positive frontier is identical. - readme scoring section updated; evaluator tests extended 26 -> 31. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bochencs · 2026-07-14T17:40:02Z

@momoway Thanks for the review! I pushed a quick fix so that invalid submissions now return a score_unbounded sentinel of −1e6 so they always rank below valid attempts, and the scale is recalibrated (target 1.20 → 1.017). Local validation tests extended and looks good! Happy to fix any other issues if needed.

momoway · 2026-07-14T21:02:49Z

@bochencs LGTM! Thanks for the fix, I'll merge the PR.

Copilot AI review requested due to automatic review settings June 17, 2026 10:06

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread 2.0/problems/lsm_compaction_policy/evaluator.py Outdated

Comment thread 2.0/problems/lsm_compaction_policy/evaluator.py Outdated

Comment thread 2.0/problems/lsm_compaction_policy/evaluator.py Outdated

Comment thread 2.0/problems/lsm_compaction_policy/readme Outdated

Add RocksDB native compaction policy task

eda84cc

bochencs force-pushed the frontiercs2/lsm-compaction-policy branch from 16dc172 to eda84cc Compare July 12, 2026 01:45

bochencs changed the title ~~Add new Frontier-CS 2.0 problem lsm_compaction_policy~~ Add RocksDB native compaction policy task Jul 12, 2026

momoway merged commit 84089cf into FrontierCS:main Jul 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add RocksDB native compaction policy task#156

Add RocksDB native compaction policy task#156
momoway merged 2 commits into
FrontierCS:mainfrom
bochencs:frontiercs2/lsm-compaction-policy

bochencs commented Jun 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joyemang33 commented Jun 19, 2026

Uh oh!

bochencs commented Jun 19, 2026 •

edited

Loading

Uh oh!

joyemang33 commented Jun 20, 2026

Uh oh!

bochencs commented Jul 12, 2026

Uh oh!

joyemang33 commented Jul 12, 2026

Uh oh!

momoway commented Jul 14, 2026

Uh oh!

bochencs commented Jul 14, 2026 •

edited

Loading

Uh oh!

momoway commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

bochencs commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

GPT-5.5 difficulty check

Judge Image

Checklist

CI Validation (for new problems)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joyemang33 commented Jun 19, 2026

Uh oh!

bochencs commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joyemang33 commented Jun 20, 2026

Uh oh!

bochencs commented Jul 12, 2026

Uh oh!

joyemang33 commented Jul 12, 2026

Uh oh!

momoway commented Jul 14, 2026

Uh oh!

bochencs commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

momoway commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bochencs commented Jun 17, 2026 •

edited

Loading

bochencs commented Jun 19, 2026 •

edited

Loading

bochencs commented Jul 14, 2026 •

edited

Loading